Parameters to evaluate a simple linear regression model: R^2 and SEE

•0•0Comment iconComment iconComment iconComment icon

This article addresses the fundamental principles of analyzing a simple linear regression model, highlighting aspects such as choice of parameters, statistical tests and interpretation of results for accurate validation.

Writer image

Translated byEditorial

Writer image

Revised byEditorial

Edit Article

In this lesson, we will explore how to evaluate whether a linear regression model is good, including performance metrics, statistical tests, and residual analysis. In addition, we will look at practical exercises with data tables to calculate regressions and interpret results.

Linear regression is a statistical technique that models the relationship between a dependent variable (Y) and an independent variable (X). But how do we know whether the model is reliable? To do this, we need to evaluate its quality using statistical indicators.

Quality Indicators of a Linear Regression

Residual Analysis

Residuals are the model’s errors:

\[ \text{Residual} = y_i - \hat{y}_i \]

For a good model, the residuals should:

1. Be random (with no clear patterns).

2. Have a mean of zero (with no systematic bias).

3. Have constant variance (homoscedasticity).

4. Follow a normal distribution (for valid inference).

How to evaluate residuals?
MethodWhat to check?
Residuals vs. Predicted PlotWhether there are patterns (curves, funnels)
Residual HistogramWhether the distribution is normal
Q-Q PlotWhether the residuals follow normality
Common problems:

- Heteroscedasticity: Non-constant variance (“funnel” plot).

- Non-normality: Skewed residuals (affects statistical tests).

- Autocorrelation: Temporal patterns (in time series).

Example of a good regression:

Well-fitted regression
Well-fitted regression

Example of a good residual:

Good residual
Good residual

Notice that here we can use concepts such as Kurtosis and Skewness to ensure that the residuals have more desirable shapes such as "Symmetric" and "Mesokurtic".

Standard Error of the Estimate (SEE)

SEE (Standard Error of the Estimate) measures the average error the model makes when predicting Y.

Formula:

\[ SEE = \sqrt{\frac{\sum (y_i - \hat{y}_i)^2}{n - 2}} \]

- The lower the SEE, the more accurate the model.

- Useful for comparing different models.

t-Test and p-Value

The t-test checks whether the slope coefficient (β₁) is statistically significant.

Hypotheses:

- \( H_0: \beta_1 = 0 \) (there is no relationship)

- \( H_1: \beta_1 \neq 0 \) (there is a relationship)

Interpretation of the p-value:

- p < 0.05: Reject \( H_0 \) (significant relationship).

- p > 0.05: Do not reject \( H_0 \) (no evidence of a relationship).

Confidence Interval for the Coefficients

The 95% CI for \( \beta_1 \) shows the range of plausible values.

- If it does not include zero, there is a significant relationship.

Coefficient of Determination (R²)

R² (coefficient of determination) measures how much of the variation in Y is explained by X.

Formula:

\[ R^2 = \frac{SQ_{\text{Regression}}}{SQ_{\text{Total}}} \]

Where:

- \( SQ_{\text{Regression}} \) = Regression Sum of Squares

- \( SQ_{\text{Total}} \) = Total Sum of Squares

Or

\[ R^2 = 1 - \frac{SEE}{SQ_{\text{Total}}} \]

Interpretation:

- 0 ≤ R² ≤ 1 (the closer to 1, the better).

- If \( R^2 = 0.75 \), it means that 75% of the variation in Y is explained by X.

Limitations of R²:

- Does not indicate causality (only association).

- Can be misleading if there are many outliers or nonlinear relationships.

Practical Example with Data Table

Let’s analyze a regression between hours of study (X) and test score (Y).

StudentHours (X)Score (Y)
1250
2460
3680
4870
51090
Steps to calculate the regression:

1. Calculate \( \hat{Y} = \beta_0 + \beta_1 X \)

2. Evaluate R², SEE, and p-value

3. Analyze residuals

Step 1: Calculate the Linear Regression

(\(\hat{Y} = \beta_0 + \beta_1 X\))

We need to find \(\beta_0\) (intercept) and \(\beta_1\) (slope).

Formulas:

\[\beta_1 = \frac{n \sum XY - (\sum X)(\sum Y)}{n \sum X^2 - (\sum X)^2}\]

\[\beta_0 = \bar{Y} - \beta_1 \bar{X}\]

Calculations:

1. Calculate the sums:

- \(\sum X = 2 + 4 + 6 + 8 + 10 = 30\)

- \(\sum Y = 50 + 60 + 80 + 70 + 90 = 350\)

- \(\sum XY = (2 \times 50) + (4 \times 60) + (6 \times 80) + (8 \times 70) + (10 \times 90) = 100 + 240 + 480 + 560 + 900 = 2280\)

- \(\sum X^2 = 2^2 + 4^2 + 6^2 + 8^2 + 10^2 = 4 + 16 + 36 + 64 + 100 = 220\)

2. Calculate \(\beta_1\):

\[ \beta_1 = \frac{5 \times 2280 - 30 \times 350}{5 \times 220 - 30^2} = \frac{11400 - 10500}{1100 - 900} = \frac{900}{200} = 4.5 \]

3. Calculate \(\beta_0\):

\[ \bar{X} = \frac{30}{5} = 6, \quad \bar{Y} = \frac{350}{5} = 70 \]

\[ \beta_0 = 70 - 4.5 \times 6 = 70 - 27 = 43 \]

4. Regression Equation:

\[ \hat{Y} = 43 + 4.5X \]

Step 2: Calculate the Predicted Values

We apply the equation to each \(X\):

Student\(X\)\(Y\)\(\hat{Y} = 43 + 4.5X\)\(Y - \hat{Y}\) (Residual)
1250\(43 + 4.5 \times 2 = 52\)\(50 - 52 = -2\)
2460\(43 + 4.5 \times 4 = 61\)\(60 - 61 = -1\)
3680\(43 + 4.5 \times 6 = 70\)\(80 - 70 = +10\)
4870\(43 + 4.5 \times 8 = 79\)\(70 - 79 = -9\)
51090\(43 + 4.5 \times 10 = 88\)\(90 - 88 = +2\)

Step 3: Calculate the Sums of Squares

1. Total Sum of Squares (\(SQ_{\text{Total}}\))

Measures the total variation of \(Y\) around its mean.

\[ SQ_{\text{Total}} = \sum (Y_i - \bar{Y})^2 \]

\(Y_i\)\(Y_i - \bar{Y}\)\((Y_i - \bar{Y})^2\)
50-20400
60-10100
80+10100
7000
90+20400

\[ SQ_{\text{Total}} = 400 + 100 + 100 + 0 + 400 = 1000 \]

2. Regression Sum of Squares (\(SQ_{\text{Reg}}\))

Measures how much of the variation in \(Y\) is explained by the model.

\[ SQ_{\text{Reg}} = \sum (\hat{Y}_i - \bar{Y})^2 \]

\(\hat{Y}_i\)\(\hat{Y}_i - \bar{Y}\)\((\hat{Y}_i - \bar{Y})^2\)
52-18324
61-981
7000
79+981
88+18324

\[ SQ_{\text{Reg}} = 324 + 81 + 0 + 81 + 324 = 810 \]

3. Residual Sum of Squares (\(SQ_{\text{Res}}\) or SSE)

Measures the error not explained by the model.

\[ SSE = \sum (Y_i - \hat{Y}_i)^2 \]

Residual (\(Y_i - \hat{Y}_i\))\((Y_i - \hat{Y}_i)^2\)
-24
-11
+10100
-981
+24

\[ SSE = 4 + 1 + 100 + 81 + 4 = 190 \]

(Note that \(SQ_{\text{Total}} = SQ_{\text{Reg}} + SSE\): \(1000 = 810 + 190\))

Step 4: Calculate \(R^2\) (Coefficient of Determination)

\[ R^2 = \frac{SQ_{\text{Reg}}}{SQ_{\text{Total}}} = \frac{810}{1000} = 0.81 \text{ or } 81\% \]

Interpretation:

The model explains 81% of the variation in the scores. The remaining 19% is unexplained (errors/residuals).

Step 5: Calculate SEE (Standard Error of the Estimate)

\[SEE = \sqrt{\frac{SSE}{n - 2}} = \sqrt{\frac{190}{5 - 2}} = \sqrt{63.33} \approx 7.96\]

Interpretation:

The model's average error is ±7.96 points in predicting the grades.

Summary of Results

- Regression Equation: \(\hat{Y} = 43 + 4.5X\)

- R² = 0.81 (81% of the variation explained)

- SSE = 190 (sum of squared residuals)

- SEE ≈ 7.96 (average prediction error)

Exercises

1. A regression between age (X) and blood pressure (Y) has \( R^2 = 0.64 \). What does this mean?

a) 64% of the variation in Y is explained by X

b) The correlation between X and Y is 0.64

c) The model is invalid

d) 36% of the data are not explained by Y

2. "The residuals form a U-shaped pattern." What does this indicate?

a) Homoscedasticity

b) Heteroscedasticity

c) Perfect normality

d) A good regression

3. Given the table below, calculate the linear regression and interpret R² and SEE.

X (Years Exp.)Y (Salary)
130,000
340,000
538,000
755,000

1. Calculate \( \beta_0 \) and \( \beta_1 \).

2. Calculate \( R^2 \) and SEE.

4. Perform a regression with the data below and discuss whether the model is good.

X (Advertising)Y (Sales)
10080
20070
30090
350110
Tip: Calculate \( R^2 \), SEE, and plot the residuals!

5. Calculate \( R^2 \), SEE, and plot the residuals of the regression below. Can we say that the regression is good? Check the Skewness and Kurtosis.

StudentHours (X)Grade (Y)Prediction (^Y)
125055
246065
368075
487085
5109095

Answer Key for the Exercises

1. a)

2. b)

Send your news

Topics

regressão parametros
Telegram logo

Hi, we have a community in Whatsapp! Come talk to us about Statistic here!